Reformatting Text Files

Introduction

One of the most common programming tasks in natural resources is to reformat a file. This happens so often be cause field crews and web sites rarely provide data in a format that software packages need. In addition, software packages do not all take data in the same format so you may need to change a file between packages. Finally, we want to reformat data to make it more accessible to humans.

In the summer of 2020, I worked on a project that pulled over 300 files on Salmon data just for the Trinity River. The data was thn QAQCed and selected files were processed with Ptyhon and put into a database for the Yurok Fisheries Department to use. This is just one of hundreds of examples where I have processed large datasets using Python.

Examples

The most common reformatting tasks are:

• Changing date formats.
  • Humans like dates like 2/15/2015 but computer databases require dates to be formatted as 2015-02-15. There are many date formats that we have to support.
• Changing coordinates
  • Coordinates can be specified as DMS, DM, DD, and other formats but computers like to work with coordinates that are in DD.
• Changing units
  • Most of the world uses metric units but the United States is still on the "English" system. Converting from metric, or ISO, units to English and back again is common.
• Scientific Names to common names
  • Changing from common names of species and back again is a common task. Also, there are different versions of scientific names that we may have to convert to a single version.